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A.  Research  Objectives 

The  research  conducted  under  the  concluding  contract  has  explored  prob- 
lems man  faces  when  integrating  information  about  his  uncertain  world,  evalu- 
ating past  decisions,  and  anticipating  future  events.  We  have  systematically 
compared  what  decision  makers  are  doing  with  Information  to  what  they  should 
be  doing,  and  we  have  begun  to  develop  tools  to  help  people  become  better 
decision  makers. 

B.  Sumrary  of  Research  Accomplishments 

1-  Probability  assessment.  Prediction,  ard  ^noice:  Systematic  biases 

in  unaided  human  judgment.  Much  is  known  about  how  probabilities  ought  to 
be  assessed — the  mathematical  constraints.  However,  little  has  been  known 
about  how  probabilities  are  assessed  by  people  whose  cognitive  constraints 
limit  their  ability  to  perform  optimally.  In  our  research,  we  have  discovered 
several  heurist.  « used  by  probability  assessors;  heuristics  which  simplify 
the  cognitive  strains  of  the  task,  but  which  can  lead  to  serious  distor- 
tions or  biases  in  che  assessments  made.  While  much  of  this  research  has 
its  origins  in  laboratory  studies,  the  implications  of  the  research  for 
practical,  applied  problems  have  been  widely  recognized. 

We  have  found  that  predictions,  too,  are  prone  to  systematic  biases. 

For  example  human  predictions  are  overwhelmingly  determined  by  specific, 
individuating  information  rather  than  base-rate  Information  reflecting 
a large  data  base.  Our  research  has  also  shown  that 
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people  making  numerical  predictions  from  numerical  cues  are  fundamentally 
non-optimal,  and  that  thrlr  ability  to  process  multivariate  information 
is  highly  limited  and  often  biased.  These  findings  lead  to  grave  concern 
about  the  adequacy  of  unaided  human  judgment,  suggesting  that  judges  should  receive 
specialized  training,  or  else  be  aided  bv,  or  sometimes  replaced  by,  computers. 

When  people  are  called  upon  to  choose  among  well-specified  alternatives 
their  preferences  may  often  be  subject  to  bias.  We  have  shown  that  in  some 
situations,  their  choices  violate  one  of  the  fundamental  axioms  of  rational 
decision  making.  Further,  we  have  shown  that  difficult  choices,  between 
alternatives  which  are  equated  in  value,  are  resolved  in  predictable,  but 
not  optimal,  wavs. 

2 . Evaluation  of  Fast  Decisions:  The  psychology  of  hindsight  and 

evaluation.  Work  performed  under  the  present  contract  has  provided  a new 
view  of  how  people  evaluate  the  past,  showing  that  people  perceive  past  events 
as  having  been  far  more  inevitable,  in  hindsignt,  than  such  events  seemed 
before  they  occurred.  Far-reaching  implications  of  this  work  are:  (a)  evalu- 

ations of  the  past  performance  of  decision  makers  may  be  unduly  harsh  in 
stating  that  "you  should  have  known  what  would  happen;  and  (b)  because  the 
past  seems  more  predictable  than  it  was,  humans  are  unable  to  learn  to 
anticipate  th-.-:  surprises  of  the  future. 

3.  Theor-,  Development.  A portion  of  the  effort  under  the  contract 
has  been  devoted  to  examining  the  adequacy  of  utility  theory  as  a normative 

and  descriptive  theorv  of  decision-making  behavior.  We  have  found  that  utility 
theory  does  not  provide  an  adequate  description  of  people's  behavior  in  the 
face  of  uncertainty.  These  results  bring  into  question  the  adequacy  of 
the  procedures  used  in  decision  analysis  to  elicit  utilities  and  probabilities. 
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C.  Highlights  of  Research  Accomplishments 

1.  Tversky,  A.  & Kahneman,  D.  Judgment  under  uncertainty:  Heuristics 

and  biases.  Technical  Report,  August,  1973.  Also  published  In 
Science.  1974,  185,  1124-1131. 

This  article  describes  three  general  heuristics  (mental  operations) 
that  are  used  to  assess  probabilities  and  to  predict  numerical  values. 

(i)  An  assessment  of  representativeness  or  similarity,  which  is  usually 
performed  when  people  are  asked  to  judge  the  likelihood  that  an  object 
or  event  A belongs  to  a class  or  process  B.  (ii)  An  assessment  of  the 
availability  (imaginabilitv,  memorability)  of  instances  or  scenarios, 
which  is  often  employed  when  people  are  asked  to  assess  the  frequency 
of  a class  or  the  plausibility  of  a particular  development.  (iii)  An 
adjustment  from  a starting  point,  which  is  usually  employed  in  numerical 
prediction  when  a relevant  value  is  available.  These  heuristics  lead  to 
systematic  and  predictable  errors  bv  both  laymen  and  experts.  The  insights 
provided  by  this  paper  should  be  a major  step  towards  improving  judgments 
and  decisions  in  situations  of  uncertainty. 

2.  Slovic,  P.  & Tversky,  A.  Who  accepts  Savage's  axiom?  Technical 
Report,  February,  1974.  Also  published  in  Behavioral  Science, 
November,  1974. 

This  paper  questions  the  acceptability  of  the  axioms  "of  rational 
decision  making.  Ir.  particular,  this  study  scrutinizes  Savage's  "Indepen- 
dence Principle"  wnich  asserts  that,  if  twc  competing  decision  alternatives 
have  a common  outcome  under  a particular  state  of  affairs,  then  the 
preference  for  one  or  the  other  alternative  should  be  independent  of 
the  value  of  that  common  outcome.  Persistent  violations  of  this  axiom 
were  observed.  Such  violations  raise  doubts  about  the  very  foundation  of 
modern  decision  analysis — i.e.,  about  the  validity  of  expected  utility 
theory.  These  violations  suggest  that  psychological  considerations 
such  as  regret  and  distaste  for  ambiguity  may  have  to  be  incorporated  into 
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utilities,  making  such  utilities  context  dependent.  This  implies  that 
utilities  measured  out  of  context  may  not  be  valid  bases  for  decision  making. 

3.  Lichtenstin , S. , Earle,  T. , A Slovic,  P.  Cue  utilization  in  a 
numerical  prediction  task.  Technical  Report,  February,  1974.  Also 
published  in  Journal  of  F.:r er imcntal  Psychology,  February,  1975. 

.’nis  stuoy  describes  a non-optional  procedure  used  by  judges  when  making 
predictions  on  the  basis  of  two  independent,  numerical  cues.  Whereas 
the  optimal  (statistical)  model  for  this  task  adds  the  separate  impacts 
of  each  cue,  almost  every  judge  ave raged  the  two  cues,  leading  to  serious 
errors  of  prediction.  This  result  demonstrates  the  need  for  special 
training  or  decision  aids  for  persons  Involved  in  making  predictions  from 
quantitative  data. 

4.  Slovic,  P.  Consistency  of  choice  between  equallv-valued  alternatives. 
Technical  Report,  November,  1974.  Also  in  press,  Journal  of 
Experimental  Psychology. 

What  happens  when  a decision  maker  is  faced  with  a choice  between 
two  alternatives  that  are  of  equal  value  to  him?  This  study  shows  that 
people  resolve  such  dilemmas  by  using  a chcice  rule  that  is  easy  to  apply 
and  easy  to  justify  to  themselves  and  others.  In  this  case,  they  consistently 
chose  on  the  basis  of  the  alternative  that  was  superior  on  the  more  important 
attribute  or  dimension,  even  though  it  was  considerably  inferior  on  a lesser 
dimension.  It  may  be  possible,  using  the  knowledge  gained  from  this  study, 
to  predict  how  conflicted  individuals  (or  perhaps  even  nations)  till 
resolve  difficult  choices. 

5.  Fischhorf,  B.  & Bevth,  R.  I knew  it  would  happen — remembered 
probabilities  of  once-future  things.  Technical  Report,  May,  1974. 

Also  published  in  Organizational  Behavior  and  Human  Performance, 
February,  1975. 

In  this  study  judges  who  had  estimated  the  likelihood  of  various 
possible  outcomes  of  President  Nixon's  trips  to  Peking  and  Moscow  were 
unexpectedly  asked  to  recall  their  own  predictions  some  time  after  the 
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visits  were  completed.  In  addition,  they  indicated  whether  or  not  they 
thought  that  each  event  had  in  fact  occurred.  Remembered  probabilities 
were  generally  higher  than  the  originally  assigned  probabilities  for  events 
believed  to  have  occurred  and  lower  for  those  which  had  not  occurred. 
Judging  bv  their  remembered  probabilities,  subjects  seldom  perceived 
having  been  very  surprised  by  what  had  or  had  not  happened,  thus  ex- 
hlbiting  an  I knew  It  all  along"  attitude.  The  practical  implications 
of  this  result  are  discussed  in  Study  7 below. 

6.  Fischhoff,  B.  Hindsight  4 Foresight:  The  effect  of  outcome 

knowledge  on  judgment  under  uncertainty.  Technical  Report, 

April,  1975.  Also  in  press,  Journal  of  Experimental  Psychology. 

Several  experiments  demonstrate  that  learning  some  event  has  happened 
increases  its  perceived  inevitability.  We  are  unaware,  however,  of  this 
effect  of  outcome  knowledge  and  tend  to  believe  that  this  inevitability 
was  apparent  in  foresight,  before  we  knew  what  happened.  In  retrospect, 
we  tend  to  believe  that  we  (and  others)  had  a much  better  idea  of  what  was 
going  to  happen  than  we  (or  others)  actually  did  nave.  Such  misperceptions 
can  seriously  prejudice  the  evaluation  of  decisions  made  in  the  past  and 
limit  what  is  learned  from  experience. 

7.  Fischhoff,  B.  Hindsight:  Thinking  backward?  Technical  Report, 

Novetn'  ,1974. 

This  report  provides  a non-technical  summary  of  Studies  5 and  6 
above.  In  addition,  it  presents  some  specific  techniques  for  evaluating 
the  quality  of  decision  making  without  prejudice  due  to  knowledge  of 
results.  These  include  (a)  educating  the  evaluators  with  regard  to 
the  existence  and  nature  of  uncertainty  in  the  past;  (b)  generating 
alternative  futures  for  past  situations;  and  (c)  farming  the  evaluation 
out  to  genuinely  foresightful  minds  (ignorant  of  the  decision  outcome). 
Examples  of  possible  hindsight  biases  in  evaluating  military,  socio- 
P°Htical,  and  personal  decisions  are  discussed  here. 
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8.  Fischhoff,  R.  Temporal  setting  and  judgment  under  uncertainty. 
Technical  Report,  April,  1975.  Also  in  press.  Organizational 
Behavior  and  Human  Performance. 

Are  probability  estimates  affected  by  the  temporal  setting  (i.e., 
oast  vs.  future)  of  the  events  being  judged?  It  has  been  hypothesized 
that  analysts  are  more  certain  about  the  likelihood  of  events  set  in  the 
past  than  about  events  set  in  the  future.  Experiments  reported  here 
show  that,  when  outcome  knowledge  is  witheld  trom  judgments  set  in  the 
pa3t,  temporal  setting  has  no  discernible  effects  on  probability  estimates. 

9.  Lvon,  D.  & Slovic,  P.  On  the  tendency  to  ignore  base  rates  when 
estimating  probabilities.  Technical  Report,  April,  1975. 

A fundamental  principle  of  the  normative  theory  of  prediction  is 
that  prior  probability,  which  summarizes  what  we  knew  before  receiving  any 
specific  evidence,  remains  relevant  even  after  such  evidence  is  obtained. 
However,  as  this  study  and  previous  work  by  Kahneman  and  Tversky  show, 
human  intuition  does  not  work  this  way.  Human  judges  deviate  from  the 
normative  model  by  relying  almost  exclusively  on  the  implications  of 
specific  evidence  (i.e.,  the  witness  said  X;  and  the  message  said  Y,  etc.) 
and  neglecting  prior  probabilities  (i.e.,  X and  Y are  very  rare  in  the 
population  of  behaviors  under  consideration).  As  a result,  intuitive 
probability  assessments  often  are  quite  erroneous.  We  show  that  experts, 
too,  are  susceptible  to  this  bias  and  it  is  apparent  that  training  and  use 
of  special  computational  techniques  must  be  employed  to  minimize  its  effects. 

10.  Fischhoff,  B.  Attribution  theory  and  judgment  under  uncertainty. 
Technical  Report  written  and  being  prepared  for  distribution,  July, 
1975. 

A central  activity  of  any  intelligence-gathering  operation  is  under- 
standing why  various  people  acted  as  they  did,  i.e.  , attributing  causes  to 
their  behavior.  The  quality  of  these  attributions  sets  an  upper  limit  on 
the  quality  of  the  interpretation  of  more  complex  (global)  events  inferred 
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from  them.  Identification  of  systematic  biases  in  attributional  behavior 
would  suggest  ways  in  which  an  intelligence-gathering  operation  could  improve 
its  performance.  In  this  paper,  the  extensive  psychological  literature  on 
attributional  behavior  is  reviewed,  revealing  a number  of  interesting  biases. 

For  example,  people  will  frequently  attribute  a given  act  to  circumstantial 
constraints  when  they  are  the  actor,  but  to  personal  traits  when  they  are 
the  observer.  In  general,  however,  the  "attribution  theory"  literature  is 
largely  devoid  of  applied  ramifications,  generally  ignoring  questions  of 
attributional  adequacy.  Ways  to  remedy  this  problem  and  to  further  our 
understanding  of  how  well  people  make  attributions  are  suggested.  Consider- 
able attention  is  given  to  how  we  may  increase  the  positive  transfer  between 
prediction  and  explanation. 

11.  Fisohhoff,  B & Lichtenstein,  S.  Don't  attribute  this  to  Reverend 
Bayes.  Report  submitted  for  journal  publication. 

Ajzen  and  Fishbein  (1975)  have  recently  offered  the  rules  of  Bayesian 
inference  as  a unifying  framework  for  research  on  attribution  processes. 

The  usefulness  of  their  theory,  however,  is  restricted  by  errors  and 
ambiguities  in  their  presentation  of  Bayesian  inference,  by  their  inade- 
quate review  of  the  empirical  evidence  regarding  the  extent  to  which 
people  follow  u.iyes'  theorem  in  intuitive  information  processing,  and  by 
their  misapplication  of  Bayes'  theorem  to  attributional  tasks.  A number 
of  these  flaws  a^e  pointed  out  and  an  alternative  manner  of  exploiting 
Bayesian  inference  is  suggested. 

12.  Slovic,  P. , Fischhoff,  B.  & Lichtenstein,  S.  Cognitive  processes 
and  societal  risk  taking.  Technical  Report  completed  and  about 
to  be  distributed,  duly,  1975. 

This  paper  explores  how  the  psychological  study  of  decision  processes 
might  help  those  who  seek  to  understand  and  improve  societal  decisions  involving 
risk.  The  discussion  is  organized  around  three  questions:  (a)  What  are 
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some  of  the  basic  policy  issues  regarding  societal  risk?  (b)  What  knowledge 
has  25  years  of  empirical  and  theoretical  research  produced  that  is  relevant 
to  these  issues?  (c)  What  more  do  we  need  to  know  and  how  might  we  acquire 
that  knowledge? 

D.  Work  in  Progress 

In  addition  to  the  studies  summarized  above,  work  has  been  proceeding 
on  many  other  studies  of  judgmental  biases  and  their  implications  for 
improving  human  decisions.  This  work  will  be  continued  and  completed  under 
our  new  contract. 

1.  A study  by  Tversky  and  Kahneman  is  attempting  to  demonstrate  the 

wo. kings  of  anchoring  and  adjustment  processes  in  the  assessment  of  subjective 
probability  distributions.  Tentative  conclusion:  Assessed  probabilities 

are  so  greatly  affected  by  response  mode  that  one  should  average  across 
several  assessment  techniques,  with  opposing  biases,  to  minimize  distortion. 

2.  Slovic  has  been  evaluating  the  effectiveness  of  various  forms  of 
computerized  feedback  in  teaching  people  to  integrate  information  in  complex 
ways  when  making  a quantitative  judgment  or  prediction.  Tentative  conclusion: 
Computerized  feedback,  which  transforms  conceptual  judgment  problems  to 
perceptual  tasks,  may  enable  people  to  perform  far  more  complex  judgment 
tasks  than  had  previously  been  thought  possible  (where  complex  means  many 
predictor  cues,  differing  in  importance,  and  each  having  a different,  often 
nonlinear,  relationship  to  the  criterion  being  judged). 

3.  Tversky  and  Kahneman  are  preparing  a treatise  on  "The  psychology 
of  regret.  This  study  will  illustrate  the  psychological  components  of 
utility  much  as  their  earlier  work  documented  the  components  of  subjective 
probability.  They  will  attempt  to  demonstrate  that  psychological  components 
such  as  anticipated  regret  must  be  incorporated  into  utilities  if  these 
utilities  are  to  be  valid  guides  to  decision. 
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4.  Lichtenstein  and  Fischhoff  have  been  studying  the  adequacy  of 
individuals  as  probability  assessors  and  the  possibility  of  developing 
aids  to  help  them  become  better  calibrated. 

5.  Slovic,  Lichtenstein  and  Fischhoff  have  been  studying  persons' 
perceptions  of  the  relative  frequencies  of  low  probability  hazards  such  as 
causes  of  death.  Results  indicate  that  people  have  consistent  subjective 
orderings  for  such  events,  but  these  orderings  often  differ  markedly  from 
the  statistical  frequencies. 

6.  Kahneman  and  Tversky  are  completing  a critique  of  utility  theory 

entitled  "Value  theory:  An  analysis  of  choices  under  risk".  This  paper  points 

out  the  inadequacies  of  utility  as  a descriptive  model  for  choice  under 
uncertainty  and  as  a normative  or  prescriptive  model  as  well.  Their  findings 
bring  into  question  the  techr Iques  used  by  decision  analysts  to  infer  subjec- 
tive probabilities  and  utilities  from  observed  preferences.  "Value  Theory" 

is  developed  as  an  alternative  to  utility  theory. 


